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ABSTRACT 

Objectives: To investigate whether a priori selection 
of patient records using unexpectedly long length of 
stay (UL-LOS) leads to detection of more records with 
adverse events (AEs) compared to non-UL-LOS. 
Design: To investigate the opportunities of the 
UL-LOS, we looked for AEs in all records of patients 
with colorectal cancer. Within this group, we compared 
the number of AEs found in records of patients with a 
UL-LOS with the number found in records of patients 
who did not have a UL-LOS. 
Setting: Our study was done at a general hospital in 
The Netherlands. The hospital is medium sized with 
approximately 30 000 admissions on an annual basis. 
The hospital has two major locations in different cities 
where both primary and secondary care is provided. 
Participants: The patient records of 191 patients with 
colorectal cancer were reviewed. 
Primary and secondary outcome measures: 
Number of triggers and adverse events were the 
primary outcome measures. 
Results: In the records of patients with colorectal 
cancer who had a UL-LOS, 51% of the records 
contained one or more AEs compared with 9% in the 
reference group of non-UL-LOS patients. By reviewing 
only the UL-LOS group with at least one trigger, we 
found in 84% (43 out of 51) of these records at least 
one adverse event. 

Conclusions: A priori selection of patient records 
using the UL-LOS indicator appears to be a powerful 
selection method which could be an effective way for 
healthcare professionals to identify opportunities to 
improve patient safety in their day-to-day work. 
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INTRODUCTION 

Within health services research, increased 
attention is focusing on patient outcomes. This 
results from the need to improve care and the 
need to reduce costs. As studies increasingly 
evaluate patient care, the need exists to identify 
adverse outcomes within patient medical 
records. This is a major challenge because 
medical records are usually extensive and 
sometimes difficult to evaluate. In the USA, 
this subject is being addressed by computer 
algorithms such as the Potentially Preventable 



Article focus 

■ Record reviewing in order to identify adverse 
events is time-consuming. 

■ A priori selection of patient records on the basis 
of unexpectedly long length of stay (UL-LOS) 
can be an efficient way to increase the chance of 
finding adverse events. 

Key message 

■ Selection of patient records with the UL-LOS and 
use of the Global Trigger Tool appear to be a 
powerful way of finding a majority of adverse 
events while limiting the number of patient 
records to be reviewed and thereby saving the 
reviewing physicians' valuable time. 

Strengths and limitations of this study 

■ This is the first study to look at the effectiveness 
^ of UL-LOS. 

^ A limitation of the study is that it was investi 
I gated in only one group of patients. Future 
P research should investigate the effectiveness of 
UL-LOS in other diagnostic groups. 



Complications and the Potentially Preventable 
Readmissions software. These algorithms use 
hospital discharge abstract data to identify 
adverse outcomes in large populations. The 
development of these algorithms has been a 
long and resource intensive process.^ The 
current research addresses this important 
subject by developing a tool for identifying 
adverse outcomes in the Netherlands. The 
research involved patients at a medium-sized 
hospital where the volume of inpatients makes 
the identification of specific patients with 
adverse outcomes a challenging undertaking. 

Furthermore, diminishing the number of 
patient-related adverse events became one 
of the top priorities for Dutch hospitals. 
A common way to achieve this is to learn from 
incidents and take action to prevent recur- 
rence. To identify the adverse events, retro- 
spective patient record review has become 
the 'gold standard' (inter) nationally. By 
retrospectively reviewing patient records. 



Ciliangir S, Borghans I, Hekkert K, etal. BMJ Open 2013;3:e003034. doi:1 0.11 36/bmjopen-201 3-003034 



1 



Unexpectedly long length of stay as tool for identifying adverse events 



healthcare professionals are able to identify adverse events 
that occurred during the care process. Several studies on 
retrospective patient record review in different countries 
have shown a wide range of incidences of adverse events, 
varying from 2.9% to 16.6% with a median overall inci- 
dence of adverse events of 9.2%.^"^ This implies that, with 
random selection from all hospital records, a healthcare 
professional has to review 6-34 records to find one adverse 
event. These results show that although record reviewing 
has been proved very advantageous in finding adverse 
events, there is an important disadvantage: record review- 
ing is very time-consuming. Although most Dutch hospi- 
tals want to analyse their patient records for adverse events 
in order to identify patient safety opportunities, many hos- 
pitals are not able to mobilise enough physicians who can 
spend many hours reviewing patient records. 

Looking for more efficient ways to organise patient 
record reviewing, we investigated how to increase the 
chance of finding adverse events. Previous research has 
shown strong relationships between adverse events and 
outcomes of quality indicators at patient and hospital 
level. For instance, one study identified a relationship 
between complications and increased mortality and length 
of stay (LOS).^^ A recent study on the USA Veterans 
Health Administration data replicated these relationships 
between adverse events and patient safety indicators of the 
Agency for Healthcare Research and Quality. Several 
other studies have indicated that adverse events often lead 
to prolonged LOS, and prolonged LOS could signal safety 
issues. Silber et aP for instance showed that LOS can 
be used to reflect how well hospitals and providers deal 
with complications and adverse events. 

The above mentioned studies suggest that negative 
results on quality indicators could be attributed to 
adverse events. In the current study, we hypothesise that 
records of patients with an unexpectedly long length of 
stay (UL-LOS) will show more adverse events. This 
patient safety indicator is already 3 years in use by Dutch 
hospitals and derived from administrative medical data. 
If so, the indicator could be used for selecting patient 
records in order to find more adverse events and save 
the valuable time of those reviewing patient records. 

To test the hypothesis that looking for adverse events 
can be performed efficiently by selecting patient records 
using UL-LOS, we conducted a pilot study with a retro- 
spective review of patient records in Tergooiziekenhuizen, 
a general hospital in the Netherlands. This article 
describes the pilot study. The results of this study might 
help hospitals organise their record-reviewing process in 
the most efficient way possible by using the quality indica- 
tor that already is available to them through existing 
registries. 

METHODS 

The quality indicator UL-LOS 

In our study, we used the quality indicator UL-LOS 2009 
to make the a priori selection. The UL-LOS is based on 



the data from the National Medical Registration (LMR) . 
The LMR contains demographical patient information, 
admission-related hospital data such as diagnosis and 
surgical procedures. UL-LOS is generated by indirect 
standardisation on three patient characteristics: age, 
primary diagnosis and the main procedure that the 
patient underwent. Age of the patient is divided into five 
classes of 0, 1-14, 15-44, 45-64 and 65 and older. For 
the primary diagnoses we used the diagnosis that led to 
the admission which includes approximately 1000 diag- 
noses classified by the ICD9 in three digits. Finally, the 
main procedure is determined by the Dutch 
Classification System of Procedures and is considered to 
depend on the diagnosis of the patient. On average it 
includes five main procedural groups. Together, these 
three parameters resulted in 5x5x1000=25 000 cells. For 
each cell the mean length of stay has been taken as the 
expected length of stay. Then the ratio between the 
actual length of stay and the expected length of stay is 
taken to calculate the UL-LOS. We define the UL-LOS 
as a LOS that is more than 50% longer than expected. 
Patients who died in the hospital are excluded. In add- 
ition, UL-LOS is a quality indicator Dutch hospitals use 
for their quality-improvement programmes and the 
Dutch Health Care Inspectorate uses UL-LOS in its 
supervision of hospital care. 

Setting 

The study was done in 2010 and 2011 at 
Tergooiziekenhuizen, a general hospital with nearly 
30 000 clinical admissions a year. We used data and 
patient records from 2009. The hospital board gave us 
permission to use the data. 

Reference groups 

To assess the impact of the indicator UL-LOS, we 
selected from 191 colorectal admissions, records with 
the UL-LOS and compared these records with the refer- 
ence group consisting of comparable patients from the 
same specialty population, who were treated at 
Tergooiziekenhuizen without a UL-LOS. 

Analysis witli tlie IHI Global Trigger Tool 

A nurse used the Institute for Healthcare Improvement 
(IHI) Global Trigger Tool to search all selected patient 
records for triggers. ^'^ The nurse that did the screening 
of triggers for this study was experienced with the use of 
the Global Trigger Tool. Triggers may contain clues for 
identifying possible adverse events. This instrument 
adapts the classification from the National Coordinating 
Council for Medication Error Reporting and Prevention 
(NCC MERP^^) Index for Categorising Errors. Although 
originally developed for categorising medication errors, 
these definitions can be easily applied to any type of 
adverse event. The IHI Global Trigger Tool was devel- 
oped to count adverse events, determine the harm to 
the patient, and whether the adverse event was the 
result of a commission. According to the IHI, only cases 
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of commission should be counted. However, we also 
counted cases of omission, as these are also a valuable 
source of possible quality improvement.^ 

Accordingly, the tool excludes the categories A to D 
from the NCC MERP Index, because these categories 
describe incidents that do not cause harm. We used the 
categories E to I, which do describe harm that may have 
contributed to or resulted in: 

► temporary harm to the patient and required interven- 
tion (category E); 

► temporary harm to the patient and required initial or 
prolonged hospitalisation (category F); 

► permanent patient harm (category G); 

► intervention required to sustain life (category H); 

► contributed to or resulted in the patient's death (cat- 
egory I). 

A surgeon and an internist-nephrologist investigated 
and looked for adverse events in the patient records in 
which the nurse had found triggers. The physicians and 
the nurse were trained extensively according to the IHI 
Trigger Tool implementation programme to make sure 
that they all worked according to the same IHI method- 
ology. The patient records were randomly divided 
between the physicians. They analysed these records in 
the same room in order to discuss difficult cases and 
make use of each other's expertise. If necessary, they 
consulted other physicians in the hospital to make their 
judgments as accurate as possible. The harm caused by 
an adverse event was categorised according to the NCC 
MERP Index as indicated above. They also classified the 
adverse events into five categories: care, operation, medi- 
cation, intensive care (IC) and other. 

A priori record selection witli UL-LOS 

We selected all records of patients with an admission for 
colorectal cancer in 2009. Patients with colorectal cancer 
are generally considered to be a homogenous population 
in terms of LOS, and are relatively vulnerable to adverse 
events.^^ We excluded duplicated records, records of pal- 
liative patients and patients who died in the hospital. 
Then we selected patient records with a UL-LOS based 
on the calculations in the LMR discharge data. A nurse 
screened all these selected records for the presence of 
triggers with the trigger tool. Patient records with triggers 
were forwarded to the physicians to be investigated for 
adverse events and the possible harm to patients. We cate- 
gorised all records on the basis of the ratio between 
actual and expected LOS, into four groups. 

1. Actual LOS equal or less than 50% longer than 
expected 

2. Actual LOS 50%— 99% longer than expected 

3. Actual LOS 100%— 199% longer than expected 

4. Actual LOS 200% or more above the expected LOS 
The last three categories together form the patient 

group with a UL-LOS. The first category is the patient 
group we call non-UL-LOS which is at the same time the 
reference group. In sum, we pursued an approach to 
the record reviewing process by first making the 



selection on the basis of discharge abstract data and 
then working directly with patient medical records. In so 
doing, we have developed an effective tool which identi- 
fies adverse outcomes directly in hospital medical 
records that are being selected with the discharge 
abstract data. 

RESULTS 

UL-LOS-based record selection 

In 2009, the hospital in our study admitted, treated and 
discharged 191 patients with colorectal cancer. From this 
group, we excluded the duplicated patient records and 
patients who were admitted for palliative care which 
resulted in 129 unique patient records. From this group, 
we selected 85 patients with a UL-LOS (66%). Screening 
by our nurse with the trigger tool revealed that 51 of these 
UL-LOS records contained one or more triggers. Thus, 
27% of 191 records remained to be reviewed by our physi- 
cians. Of these records, 43 patient records included one 
or more adverse events: 27 records contained one adverse 
event; 10 records contained two adverse events; 4 records 
contained three adverse events; and 2 records contained 
four adverse events (see figure 1). 

In table 1, we present the physicians' classification. 
The adverse events were classified mainly as operation- 
related (45%), 60% of the adverse events were consid- 
ered to have resulted in temporary harm to the patient 
and required initial or prolonged hospitalisation (cate- 
gory F). 

The reference group: non-UL-LOS patients 

Table 2 shows the number of adverse events compared 
between UL-LOS and non-UL-LOS patients. In the 
non-UL-LOS group, in 9% (4 of 44) of the reviewed 
records, at least one adverse event was found, compared 
with 51% (43 of 85) in the UL-LOS group. We also com- 
pared three categories within the UL-LOS (table 2). 

DISCUSSION 

The effectiveness of the methodology developed in the 
current research is impressive, demonstrating that a 
large majority of the records identified, contained one 
or more adverse events. With a priori selection using the 
UL-LOS indicator, we found adverse events in 51% of 
the records, compared with 9% in the non-UL-LOS 
group. By reviewing only the UL-LOS group with at least 
one trigger (66%), we found in 84% (43 of 51) of all 
the records with at least one adverse event in the colo- 
rectal patient group. Putting it another way, by reviewing 
only the UL-LOS group with at least one trigger, which 
is 40% of all patients, we found 91% of all records with 
at least one adverse event in the colorectal patient 
group. The fact that almost all records including one or 
more adverse events can be found by concentrating on 
records of patients with a UL-LOS and triggers is 
encouraging for hospitals struggling with a sparse cap- 
acity of reviewing physicians. 
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Selection of 
Non-duplicated patient records 
n= 129 



Selection 
UL-LOS 

n = 85 (66%) 



Selection 
non UL-LOS 

n = 44 (34% of 129) 



> 1 trigger 
n = 51 (40% of 129) 



No triggers* 
n = 34 (26% of 129) 



No triggers 
n = 38 (29% of 129) 



Adverse events 
n = 43 (33% of 129) 



No adverse events 
n = 8(6%of 129) 



No adverse events 
n = 2(2%of 129) 



* Only trigger: increased length of stay 

n =number of patient records 

% is of total selection of patient records 

Figure 1 Flow chart for number of patient records for each step in review process. 



> 1 trigger 
n = 6(5%of 129) 



Adverse events 
n = 4 (3% of 129) 



The percentages of records in which adverse events 
were identified in the different categories of UL-LOS 
show that the present formal quahty indicator used by 
the Dutch Health Care Inspectorate identifies most 
adverse events. Our results show a rise in the percentage 
in which at least one adverse event was found from 50% 
longer LOS onwards. However, it also rises from 100% 
onwards. A more detailed study is needed to determine 
the appropriateness of the 50% threshold. These results 
apply to colorectal cancer. Future research could also 
investigate whether this threshold is appropriate for all 
diagnostic groups or whether we need varying 
percentages. 

An interesting finding is that only 45% of the adverse 
events in a group of surgical patients such as those 
with colorectal cancer are related to the classification 



'operation'. It seems that quality of care is determined 
by the whole chain of care, not only by the quality of the 
organisation in the operating room or the professionals 
performing the operation. 

Limitations 

An important limitation of this study is that we identified 
the number of adverse events in a single specialty popu- 
lation and in only one hospital. Future research should 
investigate the validation on other populations and show 
whether identifying adverse events in more patient 
records, in more hospitals, gives comparable results. 

Another limitation is the fact that, although we chose 
to have two physicians analyse the patient records 
together, both of them reviewed different records with 
consulting each other intensively. Therefore, we could 



Table 1 Number, type and severity ratings of adverse events found in records of patients admitted with colorectal cancer' 
and a unexpectedly long length of stay 



1 



Type of adverse event 



Care IVIedical Operation IC Otiier Total 



Severity rating of 
adverse event 



E: temporary harm to the patient and 


7 


2 


3 


1 


4 


17 


25% 


required intervention 
















F: temporary harm to the patient and 


9 


4 


21 


2 


4 


40 


60% 


required initial or prolonged hospitalisation 
















G: permanent patient harm 


0 


0 


4 


0 


1 


5 


7% 


H: intervention required to sustain life 


1 


0 


1 


0 


0 


2 


3% 


1: contributed to patient death 


1 


0 


1 


0 


1 


3 


4% 


Total 


18 


6 


30 


3 


10 


67 






27% 


9% 


45% 


4% 


15% 




100% 



4 
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Table 2 Number of adverse events compared between UL-LOS and non-UL-LOS patients and within the UL-LOS categories ^ 







IN rsuorcis curiiaininy 

stt losict 1 trinnoi* 
cii iccioi 1 11 lyyci 


IN ^anu /o OTj recuras cuniaininy 


Non-U L-LOS patients 


44 


6 


4(9) 


All patients with a UL-LOS 


85 


51 


43(51) 1 


Of which patients with 50%-99% 


33 


14 


9 (27) ^ 


longer-than-expected LOS 








Of which patients with 100%-199% 


32 


22 


20 (63) 


longer-than-expected LOS 








Of which patients with 200% and above 


20 


15 


14(70) 


longer-than-expected LOS 








UL-LOS, unexDectedIv lona lenath of stav; LOS, lenath of stav. ^ 



not measure the inter-rater reliability. Our main concern 
was to organise the review process as efficiently as pos- 
sible. Therefore, we chose parallel record reviewing. 
However, further research should show whether parallel 
analysis is reliable enough compared with consecutive 
analysis, which still contends with poor reliability.^'^ 
Although current study shows that the spare time of phy- 
sicians can be saved by efficient record selection and 
Global Trigger Tool, experienced nurses can also review 
the patient records for the existence of adverse events. 
Then the physicians only have to determine the nurse's 
findings and assess the severity of harm. Such a strategy 
can save the time of physicians even more. 

The results of this study are encouraging in showing 
that hospitals can and will use quality indicators based 
on administrative data for patient safety policy. This 
type of hospital data is usually easily available without 
an extra administrative burden for hospitals. Earlier 
research has shown the reliability of using administra- 
tive data in relation to clinical data.^^ However, the reli- 
ability of indicators such as UL-LOS depends on the 
quality of coding in hospitals. Also in the 
Netherlands, the quality of administrative hospital data 
is subject to debate. If the quality of data coding in hos- 
pitals were to improve, the selection efficiency of 
quality indicators such as UL-LOS would probably be 
more accurate. The use of such quality indicators in 
combination with effective methods as Global Trigger 
Tool identify even more easily adverse events from the 
patient records.^ 

CONCLUSION 

Easily available selection methods such as UL-LOS and 
the Global Trigger Tool may be a powerful way of 
finding a majority of adverse events while limiting the 
number of patient records to be reviewed and thereby 
saving the reviewing physicians' valuable time. This 
could help hospitals to organise their patient safety 
policy as efficiently as possible. 
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